Method for adjusting listener location and head orientation within a physical or virtual space

ABSTRACT

The present invention is a system that detects the location and head orientation of a live performer on stage in front of an audience and adjusts the individual elements of the mix either by changing them at the listener&#39;s position or by feeding the location and orientation information back to the monitor mixer or a combination of both. The adjustments would include left/right panning, relative levels, equalization, transient response, reverberation levels, panning, and time delay, as well as other possible modifications to the signal so that the performer senses that he/she is actually listening to the various instruments, vocals, acoustic space and audience, in their real or virtual locations.

BACKGROUND

Many performers on stage use “in ear monitors” (IEMs) rather thanmonitor speakers. There are a number of advantages for the use of IEMs.One advantage is that the IEMs allow the performers to move around thestage and still hear their own monitor mix. Another advantage is thatwithout speakers on stage, the chance of feedback loops forming betweenthe monitor speakers and the microphones is eliminated. Anotheradvantage is that without the high levels from monitor speakers leakinginto open microphones on stage, the front of house mix is cleaner.

A common complaint about IEMs is that because the performer is presentedwith the same mix regardless of the location of the performer on stageor the orientation of the performer's head relative to other performerson stage and the audience in front, the performer feels “separated” or“isolated” from the live performance, diminishing the realism andimmediacy that performers rely upon. Also, the sonic image is all insidethe listener's head. This adds to the performer's cognitive load,speeding the onset, and increasing the amount, of listener fatigue.Listener fatigue is a temporary threshold shift (lower level signals arenot heard) and a spreading of the critical bands (hearing in noise isdiminished) The typical response to listener fatigue is to increase thesound level, thus increasing the fatigue and its ill effects. A vocalistor a musician relies on his/her ability to hear in order to perform. Asthat ability is diminished the performance suffers.

One effort to mitigate this problem has been to add ambience microphonesto the mix. But that “ambience” is still the same regardless of wherethe performer is on stage and which way the performer is facing. Anothereffort has been to vent the in-ear ear piece itself so that there isreduced isolation. This gives a greater sense of ambience but at thepossible cost of clarity of the actual output of the in-ear monitor aswell as possible phase cancellation between the ambient and electronicsignals in the monitor. In addition, there may be instruments that haveno acoustic output on stage (e.g.—electronic keyboards, electronicdrums) and therefore would not be part of the ambience mix. Thesesolutions would not give the performer any accurate sense of location onstage or proximity to other performers.

There are locating systems that track a performer's location on stagebut use that information to adjust the panning of that performer'ssignal only for the front of house mix in the PA system, not for themonitor system.

Real-time locating systems (RTLS) are used to automatically identify andtrack the location of objects or people in real time, usually within abuilding or other contained area. Wireless RTLS tags are attached toobjects or worn by people, and in most RTLS, fixed reference pointsreceive wireless signals from tags to determine their location. Examplesof real-time locating systems include tracking automobiles through anassembly line, locating pallets of merchandise in a warehouse, orfinding medical equipment in a hospital. The physical layer of RTLStechnology is usually some form of radio frequency (RF) communication,but some systems use optical (usually infrared) or acoustic (usuallyultrasound) technology instead of or in addition to RF. Tags and fixedreference points can be transmitters, receivers, or both, resulting innumerous possible technology combinations. RTLS are a form of localpositioning system, and do not usually refer to GPS or to mobile phonetracking. Location information usually does not include speed,direction, or spatial orientation.

A number of disparate system designs are all referred to as “real-timelocating systems”, but there are two primary system design elements:

ID signals from a tag are received by a multiplicity of readers in asensory network, and a position is estimated using one or more locatingalgorithms, such as trilateration, multilateration, or triangulation.Equivalently, ID signals from several RTLS reference points can bereceived by a tag, and relayed back to a location processor.Localization with multiple reference points requires that distancesbetween reference points in the sensory network be known in order toprecisely locate a tag, and the determination of distances is calledranging. Another way to calculate relative location is if mobile tagscommunicate directly with each other, then relay this information to alocation processor.

RF trilateration uses estimated ranges from multiple receivers toestimate the location of a tag. RF triangulation uses the angles atwhich the RF signals arrive at multiple receivers to estimate thelocation of a tag. Many obstructions, such as walls or furniture, candistort the estimated range and angle readings leading to variedqualities of location estimate. Estimation-based locating is oftenmeasured in accuracy for a given distance, such as 90% accurate for 10meter range. Systems that use locating technologies that do not gothrough walls, such as infrared or ultrasound, tend to be more accuratein an indoor environment because only tags and receivers that have lineof sight (or near line of sight) can communicate.

There is a wide variety of systems concepts and designs to providereal-time locating.

-   -   Active radio frequency identification (Active RFID)    -   Active radio frequency identification-infrared hybrid (Active        RFID-IR)    -   Infrared (IR)    -   Optical locating    -   Low-frequency signpost identification    -   Semi-active radio frequency identification (semi-active RFID)    -   Passive RFID RTLS locating via Steerable Phased Array Antennae    -   Radio beacon    -   Ultrasound Identification (US-ID)    -   Ultrasonic ranging (US-RTLS)    -   Ultra-wideband (UWB)    -   Wide-over-narrow band    -   Wireless Local Area Network (WLAN, Wi-Fi)    -   Bluetooth    -   Clustering in noisy ambience,    -   Bivalent systems    -   Depending on the physical technology used, at least one and        often some combination of ranging and/or angulating methods are        used to determine location:    -   Angle of arrival (AoA)    -   Line-of-sight (LoS)    -   Time of arrival (ToA)    -   Multilateration (Time difference of arrival) (TDoA)    -   Time-of-flight (ToF)    -   Two-way ranging (TWR)    -   Symmetrical Double Sided-Two-Way Ranging (SDS-TWR)    -   Near-field electromagnetic ranging (NFER)

There are commercial products that use various means to detect headorientation such as Waves NX or Klang, but they do not use any means todetect location, diminishing their usefulness in any application wherethe listener and/or the signal sources are moving.

Other systems have used tracking devices worn by the performers forlocalizing the performers on the stage. Seehttps://ubisense.net/en/news-events/news/ubisense-and-outboard-deliver-vocal-localisation-solutions-tokyos-new-national-theatre-and-finlands-national-operaHowever, this system does not assist the performers themselves and doesnot track the orientation of the heads of the performers.

OBJECTS OF THE PRESENT INVENTION

It is an object of the present invention is to better correlate thesonic image with the performer's visual image;

It is another object of the present invention to reduce listenerfatigue;

It is another object of the present invention to vary the mix based onthe location of the performer;

It is another object of the present invention to vary the mix based onthe head orientation of the performer;

It is another object of the present invention to have the sonic image infront of the performer;

It is another object of the present invention to improve virtual realitysystems.

These and other objects will be evident from the review the followingspecification and drawings.

SUMMARY OF THE INVENTION

The preferred embodiment of the invention is a system that detects thelocation and head orientation of a live performer on stage in front ofan audience and adjusts the individual elements of the mix either bychanging them at the listener's position or by feeding the location andorientation information back to the monitor mixer or a combination ofboth. The adjustments would include left/right, front/back and up/downpanning; relative levels; equalization; transient response;reverberation levels, direction, time delay; as well as other possiblemodifications to the signal so that the performer senses that he/she isactually listening to the various instruments, vocals, acoustic spaceand audience, in their real or virtual locations.

The location detection system has a detection system on or near thestage. Each performer would wear one or more remotely readable devices(such as a RFID tag) with unique digital identities. The detectors coulduse relative signal strength, time delay difference, triangulation,zoning algorithms or other methods or combination to locate theperformer in the performance area.

Orientation information would be generated by a means such as amagnetometer or gyroscope to detect the performer's head relative toknown horizontal and vertical orientation. The detection device could beplaced on or near one of the ear pieces in order to follow theperformer's head movements. That information would be sent to a beltpack receiver and make adjustments in the pack or be sent back to themonitor mixing desk for it to make adjustments or a combination of both.

A “reference” mix would be created based on where the performer wouldmost often be standing and facing the audience. As the performer movedaround the stage the components of the mix would be modified by level,delay, equalization, transient response, reverberation, etc. Forexample, as the performer moved closer to an instrument, that instrumentwould be increased in level, have less delay time, have greater highfrequency equalization, and greater transient response. As the performermoved away from the instrument the level of the instrument would bereduced, the delay time increased, and has less high frequencyequalization and transient response. The ratio of direct signal toreverberation as well as the timing of the reflections could also bemodified based on the proximity of the performer to an instrument.

As the performer turned his/her head, that is, changed orientation, thestereo mix would be modified. For example, as the performer turnshis/her head from facing the audience to the right, the stereo imagewould shift so that the audience was more in the left ear and theinstruments were more in the right ear. The mix could also be modifiedto account for head related transfer function (HRTF). In addition tomodifying the stereo mix to indicate right and left, it is alsoenvisioned that the orientations of up and down and back in front wouldalso be modified, providing the performer with a complete orientation ofthe sound in space.

The apparatus that receives the location and orientation informationwould be capable of making all the adjustments within preset limits andalso be able to revert to a reference mix if the location or theorientation system fails.

The sonic image when listening to headphones is almost always inside thelistener's head. There are techniques which bring the image out of thelistener's head, but only to side to side and behind the head, not infront. One of the aims of the invention is to better correlate the sonicimage with the performer's visual image.

Each individual has “learned” to locate a sonic image based on his/herown unique combination of pinna shape and head size, necessitatingcustomized measurement for each individual. One method for makingindividualized headsets is to use miniature microphones in each ear andmeasure the response for different input locations (Smyth Research).Another method would be to have a general setting and through successiveapproximation adjust the setting so that the individual correlates thesonic image to a real or virtual visual image.

Other applications for this technology include, but are not limited to,education and training systems, virtual and augmented reality displays,games, museums, and amusement park rides.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view of the prior art stage and mixers.

FIG. 2 is a view of the stage showing the performers and theirorientation facing the audience.

FIG. 3 is a view of the stage showing the performers with performer 1moving to stage left and facing the audience.

FIG. 4 is a view of the stage showing performers and their orientationwith performer 1 at the stage left, facing the back right of the stage.

FIG. 5 is a block diagram of the Monitor Mixer and computer controllingthe input for the performers' microphones and the output to the variousperformers' headphones and/or speakers.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the prior art use of the monitor and house mixers and theirrelationship to the performers and the audience.

On the stage 200 are located the performers 102-108, including, forexample only, the drummer 102, keyboard player 104, backup vocalists 106and the main performer 108. Each of the above performers has associatedmicrophones 103, 105, 107 and 109 to pick up the performers' voicesand/or instruments. Wired microphone signals are split through atransformer not shown and wireless microphone signals are split afterthe receiver. One leg of the split is sent to the monitor mixer 110 andone signal is sent to the house mixer 112. The monitor mixer 110 istypically located on the side of the stage and is operated by a consolemonitor mixer person, not shown. The monitor mixer 110 receives an inputfrom each of the performers and/or the performers' instruments and theperson operating the monitor mixer 110 can control the volume and toneof each of the inputs from the performers, as well as otherwise alterthe input signal. For example, the monitor mixing person can increasethe sound of the drums or the keyboard or the vocal, or change the tonefor one, or any combination of each of them. The monitor mixer 110 sendsits outputs to the headphones 111, 113, 115, and 117 worn by each of theperformers 102, 104, 106 and 108 and/or to the speaker 111 a, 113 a, 115a, and 117 a associated with each performer 102-108 either by hard wireor by wireless connection. The outputs from the monitor mixer 110 arethus heard by each of the performers.

The monitor engineer will create an individual mix for each performerbased on that performer's preferences. Each mix is then sent to eitherthe monitor speakers located near each performer, or via RF toheadphones worn by each performer. There are performers who are in afixed position, such as a drummer, who may use just speakers. Performerswill move around the stage would use just the wireless earphones. Someperformers demand to have both.

The house mixer 112 also receives the output signal from each of theperformers' microphones 103-109. The house mixer 112 is typicallylocated towards the rear of the audience, not on the stage. The outputof the house mixer 112 is amplified and sent to the speakers 120 and 122that the audience hears. The number and location of the speakers 120 and122 are selected for optimizing the volume and quality of what is heardby the audience. The performers 102, 104, 106 and 108 do not hear verymuch of the output of the house speakers. The operator of the housemixer 112 modifies the output of the house mixer to maximize theaesthetic sound heard by the audience. For example, the house mixer mayincrease the volume of the performer while reducing the volume of thevocal background 106. The house mixer can also increase the overallvolume and balance of the house speakers.

Referring to FIG. 2 the localization system of the present invention isshown. Performers 1 through 7 are all equipped with active radiofrequency identification tags and stereo in-ear monitors and/orassociated speakers (not shown). Anchors 10 through 14 receive signalsfrom the tags. Anchors send the received signals to the computer 30which analyzes the relative time delay between the signals and locateseach of the tags on the stage. Fixed locations such as ambiencemicrophones 20, 21 are set in the computer.

Computer 30 sends localization information to the monitor mixer 40. Themonitor mix for each performer is automatically adjusted based on thelocalization information.

Referring to FIG. 3, the localization system of the present invention isshown where Performer 1 moves to stage left. As Performer 1 moves, thelocalization system sends the signals to the computer 30 which computesthe new relative locations between each performer and fixed locations.

The computer 30 sends the localization information to the monitor mixer40. The monitor mixer 40 adjusts the mix for each performer as follows:

In comparison to FIG. 2, the panning of the audio signal from Performer1 would shift to the left in the monitor mixes for Performers 2 through7. The level of the audio signal from Performer 1 would increase in themonitor mixes for Performers 2 and 3, decrease for Performers 4,5,6 and7. The delay time for the audio signal from Performer 1 would decreasefor Performers 2 and 3, and increase for Performers 4,5,6, and 7. Theseadjustments in combination with other adjustments (reverberation,equalization, head related transfer function, etc.) would be made in themonitor mixes for Performers 2 through 7 so that their aural perceptionof where Performer 1 is at any particular moment will match their visualperceptions.

The panning of the audio signals from Performers 2 through 7 and fromAmbience Microphones 20 and 21 would shift to the right in the monitormix for Performer 1. The audio signals from Performers 2 and 3 and fromAmbience Microphone 20 would increase and the audio signals fromPerformers 4,5,6 and 7 and Ambience Microphone 21 would decrease in themonitor mix for Performer 1. The delay times of the audio signals fromPerformers 2 and 3 and ambience microphone 20 would decrease and thedelay times of the audio signals from Performers 4, 5, 6 and 7 andAmbience Microphone 21 would increase. These adjustments along withother adjustments (reverberation, equalization, head related transferfunction, etc.) would be made in the monitor mix for Performer 1 so thatperformer's aural perception of where Performers 2 through 7 are at anyparticular moment will match his/her visual perception.

Referring to FIG. 4, the head orientation system of the presentinvention is shown as Performer 1 turns his/her head to the right. Anelectronic compass located in or near one of the two in-ear monitorsdetects the change of orientation. The orientation information is sentfrom Performer 1 to the computer 30 through a Bluetooth connection. Thecomputer sends the information to the monitor mixer. The monitor mixerautomatically adjusts monitor mix for Performer 1 as follows:

The panning of the audio signals from Ambience Microphones 20 and 21 andPerformers 6 and 7 would shift to the left. Audio signals fromPerformers 4 and 5 would be centered. Audio signals from Performers 2and 3 would shift to the right. Head related transfer functionadjustments would be applied on audio signals based on the neworientation of Performer 1. These adjustments will aid in makingPerformer 1's aural perception match his/her visual perception.

Referring to FIG. 5 Monitor Mixer 40, the Monitor Mixer 40 has thestandard audio inputs and outputs. In the event that the location and/orthe orientation systems fail, the audio mixes would still be sent to theperformers. In addition to the standard features found on monitormixers, the Monitor Mixer 40 would have the added features of adjustingtime delay, stereo balance, equalization, reverberation and transientresponse for every input to each mix. The Monitor Mixer 40 would alsohave the feature of adjusting HRTF, and stereo balance on each output.

The Anchors 10-14 send their data to the Computer 30 which analyzes thedata to determine the location of each performer. The Computer 30 sendscontrol signals to the Monitor Mixer 40 which is equipped to receivethese signals. The Monitor Mixer 40 makes the appropriate audioadjustments to conform the audio image to the location of the audiosource relative to the location of the audio output.

The orientation data generated by the compass in the performers' in-earmonitors are sent to the Computer 30. The Computer 30 generates controlsignals and sends them to the Monitor Mixer 40. The Monitor Mixer 40makes the appropriate adjustments to the individual audio outputs toconform to the head orientation of each performer.

What is claimed:
 1. A method of providing a monitor mix from a soundmixer to at least one live performer wearing headphones and moving andsaid live moving performer having an output signal associated with saidmoving performer, the inputs to the sound mixer being from a pluralityof sources including the output signal from the live performer, thelocation of at least some of said sources capable of beingelectronically determined, comprising the steps of: adjusting at leastsome of the inputs associated with said sources within the sound mixerbased on the relative location of at least some of those sources to eachother and transmitting the output from the sound mixer to the headphonesof said live performer.
 2. The method of claim 1 in which the sound mixsent to the performer is dependent on the orientation of the performer'shead.
 3. The method of claim 1 in which the headphones are in earmonitors (IEMs).
 4. A method of providing a sound mix from a sound mixerto at least one live performer who is moving, the input of the soundmonitor mixer being the elements of a plurality of inputs and the movingperformer generating an output signal associated with the movingperformer, and having a location identification tag and an in earmonitor (IEM), comprising the steps of: providing the sound monitormixer with each of the inputs, including the output from the outputsignal associated with the live performer moving and the output from atleast one location identification tag for at least one performer movingso as to locate the performer as said performer moves; andelectronically configuring the sound monitor mixer to detect changes inthe inputs to the sound monitor mixer dependent on the location of eachinput, and the sound monitor mixer configured to generate an output tothe in ear monitor (IEM) of at least said moving performer, said outputbeing each separate element of the mix dependent on the location of eachperformer.
 5. The method of claim 4 in which the sound mix sent to themoving performer is dependent on the orientation of the movingperformer's head.
 6. A sound monitor mixer for transmitting a sound mixto at least one live performer who is moving wearing a locationidentification tag and an in ear monitor (IEM), the sound monitor mixerconfigured to modify a plurality of elements dependent on the locationof said at least one moving performer said moving performer having anoutput signal associated with said moving performer, said output signalbeing sent to the mixer, the output of said monitor mixer being sent tosaid in ear monitor (IEM) worn by said at least one live movingperformer.
 7. The sound mixer of claim 6 in which the output of thesound mix is dependent on the orientation of each performer's head.
 8. Amethod of providing a monitor mix from a sound mixer to at least onemoving live person wearing headphones and having an output signalassociated with said live moving person, the inputs to the sound mixerbeing a plurality of sources, including the output signal associatedwith said live moving person, the location of at least some of saidsources capable of being determined, comprising the steps of:electronically adjusting at least some of the inputs associated withsaid sources within the sound mixer based on the relative location of atleast some of those sources to each other and transmitting the outputfrom the sound mixer to the headphones of said live moving person. 9.The method of claim 8 in which at least one of the sources is associatedwith a virtual image.
 10. The method of claim 8 in which the sound mixsent to the live person is dependent on the orientation of the liveperson's head.
 11. A method of providing a monitor mix from a soundmixer to at least one live moving person wearing headphones for use withvirtual reality inputs, an output signal associated with said livemoving person, the inputs to the sound mixer being a plurality ofsources including the output signal associated with the live movingperson, comprising the steps of: adjusting at least some of the inputsassociated with said sources within the sound mixer based on therelative location of at least some of those sources to each other andtransmitting the output from the sound mixer to the headphones of saidlive moving person.
 12. The method of claim 11 in which the sound mixsent to the live moving person is dependent on the orientation of thelive moving person's head.